For a long time, electric organs have incorporated features that automate some aspect of playing music to make it easier for a novice musician to play music that sounds pleasing. These devices can play a rhythm track, or play an entire accompaniment selected by a single key. They can also provide more control by allowing the player to play the significant notes of the accompaniment, while automatically xe2x80x9cfilling inxe2x80x9d and voicing the chords appropriately. However, these devices typically require at least some practice on the part of the player, and are therefore not suited to casual or one-time use by non-musicians.
Other devices are similar in function, but are designed for use by professional musicians. These typically are set up as MIDI sequencers with advanced controls that can be manipulated from a variety of input devices. A performer can use them to automate the generation of accompaniment music, or even whole melodies, while still allowing the flexibility to alter the performance while it is happening. These devices allow a single performer, such as a nightclub entertainer, to play nearly arbitrary requests from the audience, and still maintain a full sound, while not requiring an entire band of musicians. However, the complexity of control of these devices, and the potential for error that they introduce, take them out of the realm of entertainment machines designed for non-musicians.
Music learning devices have been created that allow a student to play along with either written or pre-recorded music, measure some aspect of the student""s performance, and provide feedback on the quality of the performance. These devices typically run on a general-purpose computer, and use input controllers that either closely mimic the operation of an actual musical instrument, or are actually the instrument. By definition, they are designed for non-musicians to use (at least for the initial lessons), but they usually require some commitment of effort, and are not really entertaining enough to be attractive for casual or one-time use. In addition, they typically are not set up to sound good when the player plays incorrectly, since the point is to educate the student to play correctly.
Another professional device exists that uses the chord structure of the music to set up the keyboard so that it only plays notes that are part of the scale currently in use. This allows the player to improvise against the music more easily. A consumer version of this product exists that is implemented on a general-purpose computer. However, without any musical training, the improvisations that a player creates tend to be either monotonous or bizarre.
Some modern forms of music are based primarily on sampling, where short audio segments are played in rhythm to a backing track. As a result, some toys and other consumer products exist that allow non-musicians to select and play samples while a backing track is playing. Once again, without any musical training, the rhythmic improvisation produced by a novice tends to be fairly monotonous.
A device exists that allows non-musicians to control a melody that is automatically generated and played along with a pre-recorded accompaniment. By using a joystick or mouse input device, the player can control the general pitch (higher and lower) of the melody, as well as the density of notes in it. This device, which is implemented using a general-purpose computer, does not provide the player with the immediate tactile feedback that creates the illusion of playing an actual musical instrument.
An entertainment device exists that provides a display for a non-musician to follow and strum a guitar-like instrument or play a drum-like instrument. As a result, the device generates a musical part that is played along with a pre-recorded accompaniment. The player is rated on the accuracy of the performance, and the rating is used to control various responses of the machine. This device is again implemented using a general-purpose computer. However, this device uses a single part for an entire song, making it difficult to adjust the part dynamically to adapt to the skill of the player. In addition, the musical part is created as a single unit, making it relatively difficult and expensive to add new songs to the repertoire.
Several popular Japanese arcade games also provide a display for a non-musician to follow, and use a simple input device to play a generated musical part along with a pre-recorded accompaniment. These games are very similar to the entertainment device just described, and subsequently, include the same shortcomings.
Multiple musicians at disparate geographic locations have played together using computer networks to transmit performance information to each other. However, this has been done by musicians in constrained environments using low latency networks.
The present invention enables a non-musician to produce reasonable music without any prior training. The invention relates to systems that allow individuals with limited or no musical experience to play along with pre-recorded music in an entertaining way. The invention allows a complete novice to use an extremely simple input device to play a part that fits in well with a harmonious background music part. The invention is instantly accessible to a beginner, and produces a reasonable-sounding part regardless of the skill of the player. The present invention provides the player with a guide to follow, and organizes the guide in the same conceptual way that music is organized. The guide of the present invention gives the player something to follow, and the automated note selection of the invention avoids the monotony that occurs in sampling devices when a player repeatedly selects the same sample.
In addition, the present invention contains a display that provides guidance to the player rather than relying on the player""s ability to improvise. The present invention represents the part of the player as segments that are dynamically composed as the song is playing. This allows various parameters of the player""s part (such as difficulty) to be adjusted during play without degrading the quality of the part. It also allows parts for new songs to be quickly and easily composed using the library of existing segments. The present invention also allows non-musicians to play together using a public network with high and/or variable latency characteristics.
A system and method to allow a person with no formal music training to play along with an existing musical song provides an entertaining experience for non-musicians who nonetheless have an interest in and enjoy music. The system defined here uses any computing device capable of generating musical tones and acting in response to input from a user. The process used to define the part that is played by the non-musician player is very similar to the process used to compose music, and as a result, can be manipulated as the song progresses to produce interesting variations of the part.
The computing device provides the user with a multimedia (sound and video) presentation of a musical performance. In addition, it uses algorithmically generated graphics to present the user with an intuitive display indicating when the user should be playing a rhythmic passage to go along with the musical performance. Following this display, the user manipulates one or more input peripherals that are designed to capture rhythmic actions such as tapping one""s fingers, hitting with a stick, tapping one""s feet, moving one""s body, singing, blowing into a tube, dancing, or strumming taut strings. These actions are converted into a series of time-based signals that are transmitted to the computing device, which then algorithmically determines a set of musical tones to play in response to the actions. These musical tones fit in with the musical performance, and since they are played at the same time as the actions of the user, the user perceives that those actions are creating the musical tones. This provides the illusion that the user is playing along with the musical performance.
Since the computing device can have an interface to a computer network, the system can be used to implement interaction with multiple players, analogous in many ways to a band formed with individual musical instruments. In situations where the players are physically located near each other, a local area dedicated network with low latency is used, the multiple computing devices are synchronized, and the resulting synthesized parts can be heard by all players in a true cooperative xe2x80x9cbandxe2x80x9d. In situations where the players are geographically disparate, a wide area public network is used. When the latency is high, the individual players cannot be synchronized, but since they cannot hear each other, this is less important. The characteristics of each of the players"" actions are transmitted to all other players with relatively low bandwidth, and the actual result of all the players working together is synthesized for each player by their individual computing device. The actual performance is also recorded and distributed so that each player can review it and discuss it after the fact.
The display indicating what should be played is loosely based on standard musical notation, but the present invention simplifies it by displaying each note as a bar, with the length of the bar indicating the duration of the note. One indicator moves from bar to bar, showing which note the user should be playing. Another indicator moves along each bar, showing how long ago the note was played, and also showing how much time is left until the next note must be played. This display is very intuitive and simple to follow, and lends itself well to many adaptations in presentation to keep it interesting and fresh for the player.
When the player plays a note, the computing device uses a sound synthesis unit to generate a musical tone. The selection of which tone to generate is done by a stored representation of the player""s performance. This stored representation uses a structure that models the way musicians actually think about musical performances. It is a hierarchical description, corresponding to the decomposition of a song into units such as sections, phrases, measures, and notes. It has a mechanism for describing repetition, so that constructs such as repeated verses are conveniently specified. It can describe tempo change and key modulation, independent from the song structure and decomposition. It has a way to indicate multiple possibilities for the same unit of the song, in much the same way that musical improvisation typically consists of organizing pre-defined patterns into an interesting overall performance.
Since the computing device has information about both what the user is supposed to play and what the user is actually playing, it can algorithmically generate information about how well the user is playing. By using the accuracy of the player""s performance, in conjunction with a scoring algorithm, to generate a score, he computing device drives interactive feedback to indicate how well the player is playing. This measurement can be based on both the rhythmic accuracy of the performance as well as the accuracy of playing the correct selection of multiple input peripherals as indicated on the display. The correct selection of multiple input peripherals can be the correct tones played by a user on an input peripheral, for example. The device also uses this score to drive the decisions made by the note generation mechanism, so that the difficulty and variety of the parts available to the player increase as the player improves. The score is also used to drive decisions on a larger scale, such as what options the player has in terms of the available songs or the scenes that can be accessed in a game application.
The scoring mechanism is important for computer network implementations of multi-player applications. It is the fundamental mechanism for competition between multiple players, since it provides an objective measure for comparison. It also provides the mechanism for overcoming network latencies. The scoring mechanism computes higher order statistics of the player""s performance relative to the guide, which are sent across the network and used to drive a predictive model of the player""s performance. In this way, in a high latency network, each player does not hear the exact performance of the other players, but does hear a xe2x80x9crepresentativexe2x80x9d performance that gives nearly the same score as the actual performance. Later on, after the entire song has been performed, the actual combined performance is available to all players for review.
The present invention is ideally suited for use in game applications in several ways. These are described here.
The scoring mechanism is vital for a game. It allows players to compete, either with other players, their prior scores, or virtual (computer-generated) characters. It also allows immediate feedback (visual, auditory, touch, and even other sensory feedback) on the player""s performance. For example, a crowd can react with varying amounts of cheering or booing depending on the score. Finally, aggregate scores are used to drive major decision points in a game. For example, a game that is organized as several xe2x80x9clevelsxe2x80x9d will not allow the player to proceed to the next level until a certain score is attained, and higher scores are required for later levels.
The graphical display showing the user what to play is also well suited for game applications. Its constantly changing nature and composition of simple discrete graphic elements are characteristics of xe2x80x9cstatusxe2x80x9d displays that are part of nearly every game. In addition, these same elements lend themselves perfectly to alternate graphical representations that are more integrated with the game. For example, the bars could be represented as three-dimensional solids lined up in a row, and the indicator for the note that was last played could be represented by a character standing on the bar (the character would jump from bar to bar as notes were played). The indicator moving along the bar could be represented by the next bar moving down alongside the current bar, so that the player would attempt to make the character jump from one bar to the next when the tops of the two bars are even.
The ability of the present invention to incorporate many different kinds of input peripherals increases its attractiveness for arcade game implementations. Recent arcade games tend to use novel input devices as a distinguishing feature. Since the actual amount of information required from the peripherals is about the same as that provided by a push-button, a large variety of robust and inexpensive peripherals will work with the system.
The capability to actively use input from several players, either closely located or widely separated, is rapidly becoming a critical factor in the utility of technology for game applications (and other entertainment products as well). The rapid acceptance of the Internet has made multi-player gaming nearly a requirement for new games. In addition, more and more arcade games have multi-player stations as a distinguishing feature. The present invention addresses all of these issues, by providing applications for wide area networks as well as local area networks, high latency networks as well as
The ability to generate different parts for the user to play is extremely important for the xe2x80x9creplayxe2x80x9d value of a game application. In both arcade and console games, a high premium is placed on games that get players to come back and play the game again many times. By representing the player""s performance as a hierarchical structure with options and repetition in the hierarchy, the present invention provides nearly unlimited variety in the parts played by the player, in a way that makes sense musically and is logical to the player. This variety avoids a problem where the player ends up doing the same thing over, and also allows the player to have some control over what happens, opening up the exciting world of musical improvisation (in a limited but very real sense).
The ability to modify the parts played by the user dynamically is an even further extension that adds to this xe2x80x9creplayxe2x80x9d value. Since the computing device can select alternate parts in the hierarchy for the player to perform, this decision can be based on how well the player is doing, and the game will then actively respond to the player""s skill level. By getting more difficult at a rate that makes sense to the player, the game encourages additional play to master the increased difficulty.
In this way, the invention provides an enjoyable experience to non-musicians, allowing them to play along with music without additional talent or training. The principles of the invention can be extended in many ways and applied to many different environments, as will become apparent in the following description of the preferred embodiment.
A preferred embodiment of the invention relates to a music system having a peripheral, a hierarchical music data structure that represents the music to be played by a user, a digital processor and recorded music data that forms the accompanying music to which the user plays. The peripheral generates a signal in response to activation of the peripheral by a user. The digital processor receives the signal from the peripheral and drives an audio synthesizer based upon the signal.
The hierarchical structure can include at least one structural component and at least one pattern. The at least one structural component can include a plurality of alternative structural components while the at least one pattern can include a plurality of alternative patterns. The alternative structural components and the alternative patterns can include a plurality of difficulty levels. These difficulty levels can include a first difficulty level and a second difficulty level where the second difficulty level is more difficult that the first difficulty level.
The system can include a synchronizer that synchronizes the digital processor to the recorded music data. The music system can also include a scoring algorithm to generate a score based upon the correspondence between the signal generated by the user""s activation of the peripheral and the music represented by the hierarchical music data structure. This score is then used to activate a corresponding difficulty level. Alternately, a randomization algorithm can be used to determine the difficulty level within the music system.
The music system can also include a modification data structure that can be used to adjust a tempo within the hierarchical music data structure or to adjust a musical key within the hierarchical music data structure.
The music system can include a display for guiding a user in activating a peripheral device corresponding to the hierarchical music data structure. The display can include a first axis showing successive notes within the hierarchical music data structure and a second axis corresponding to the duration of notes within the hierarchical music data structure. The display can also include a first indicator that increments along the first axis to indicate to a user the note within the hierarchical music data structure to be played and a second indicator that moves along the second axis to indicate to a user the duration of the note within the hierarchical music data structure to be played.
The music system can include a local area network or a wide area network allowing for connection of a plurality of music systems. The system having a wide area network can include a statistical sampler and a predictive generator, the statistical sampler generating n-th order statistics relative to activation of the peripheral. The statistics are sent by the wide area network to the predictive generator that generates a performance based on the statistics from the statistical sampler, independent of the latency of the network. The system can also include a virtual peripheral connected to the predictive generator, such that the predictive generator drives the virtual peripheral to generate a performance. A broadcast medium can be used for transmission of recorded music data over the wide area network.